Panoramic image production

ABSTRACT

A method of assembling a panoramic image from a plurality of frames in a camera, the method comprising: (a) acquiring a video stream using a camera; (b) selecting an initial frame from the video stream for capture as a still image; (c) further selecting additional frames; and (d) stitching the additional frames to the initial frame in the camera to create a panoramic image.

RELATED APPLICATION

This application is related to a co-pending application Ser. No.11/826,772, entitled “METHOD FOR CONSTRUCTING A COMPOSITE IMAGE” by NoamSOREK and Orna BREGMAN-AMITAI

FIELD OF THE INVENTION

The present invention relates to production of panoramic images, forexample using a camera installed in a mobile phone handset.

BACKGROUND OF THE INVENTION

Digital cameras with panoramic photography capabilities are commerciallyavailable. Mobile telephones with built in digital cameras are alsocommercially available. Typically, editing and/or manipulation solutionsfor panoramic images are performed on a separate computing device notincorporated into the camera or phone handset.

In general, panoramic photography refers to production of photographicimages with exceptionally wide fields of view. Panoramic imagestypically capture a field of view comparable to, or greater than, thatof the human eye, about 160° by 75°. Maintenance of detail across theentire picture contributes to a perceived image quality. Panoramicindicates that the image offers an unobstructed or complete view of anarea. In many cases, panoramic images are presented as a wide strip.

Production of a panoramic image often involves capturing overlappingphotographic frames and matching, or “stitching” the frames together attheir overlapping edges.

The following patents and applications are generally indicative ofcurrent panoramic technology in “stand alone” cameras. The list does notpurport to be exhaustive.

U.S. Pat. No. 5,138,460 by Egawa describes an apparatus for formingcomposite images. Egawa's apparatus includes a display device fordisplaying an image to be photographed such as a portion of an object, amemory device for storing data concerning an image which has beenalready photographed such as a previously photographed portion of theobject, and a control device for enabling the display device tosubstantially simultaneously display both the image to be photographedand the image already photographed and stored as data in the memorydevice. The disclosure of this patent is fully incorporated herein byreference.

U.S. Pat. No. 5,907,626 to Toklu et al describes method for motiontracking and constructing a mosaic of video objects as well as a methodfor synthetic object transfiguration from a mosaic. According to Toklu,a 2-D triangular mesh is employed to represent a video object, whichpermits to describe the motion of the object by the displacements of thenode points of the mesh, and to describe any intensity variations by thecontrast and brightness parameters estimated for each node point. Usinga temporal history of the node point locations, the nodes of the 2-Dmesh are tracked even when they become invisible because ofself-occlusion or occlusion by another object. By adding new nodes orupdating the 2-D triangular mesh, any uncovered parts of the videoobject are incrementally added to the mosaic. A finite number of viewsrepresenting the constructed mosaic are selected and used for synthetictransfiguration of a replacement object in place of the original one.The disclosure of this patent is fully incorporated herein by reference.

U.S. Pat. Nos. 5,999,662 and 6,393,163 to Burt et al. each describe asystem for automatically generating a mosaic from a plurality of inputimages. The system sequentially executes an image alignment process, anediting process, and a combining process such that, from a sequence ofimages, the system automatically produces a seamless mosaic for variousapplications. The mosaic based display system permits a system user todisplay, manipulate and alter a mosaic. The mosaic based compressionsystem exploits the temporal and spatial redundancy in image sequencesand efficiently compresses the image information. The compression systemcan be used for compressing image information for storage in a storagedevice or can be used for compressing image information for transmissionthrough a band-limited transmission channel. The disclosures of thesepatents are fully incorporated herein by reference.

United States Patent Application 20040189849 by Hofer describes apanoramic sequence guide that can be used to guide a user in composingand capturing images to be used to create a composite panoramic image.In use, the guide tracks a viewed scene, determines the proper positionof a composition guide relative to the viewed scene, and displays acomposition guide on a live-view screen that provides an indication ofwhat portion of the viewed scene is to be captured. The displayedcomposition guide is fixed to the viewed scene such that the guide moveswith the viewed scene in the screen. The disclosure of this applicationis fully incorporated herein by reference.

Korean patent 0286306 to Choi describes a method of panoramicphotography which uses a digital still camera to easily connect picturesto construct a panoramic picture using an LCD monitor. The disclosure ofthis patent is fully incorporated herein by reference.

United States Patent Application 2004/0228544 by Takaaki et al.describes an image processing method and apparatus for generating apanoramic image in which the hue or brightness discontinuities generatedat the seams of source images are reduced. Upon compositing sourceimages having an overlap portion, the hue or brightness of each sourceimage is converted using a reference image which has overlap portionswith both the source images to be composited and the converted imagesare composited, thereby reducing the hue or brightness discontinuitiesgenerated at the seam of images. The disclosure of this application isfully incorporated herein by reference.

U.S. patent applications 2006/0062487, 2006/0018547, 2005/0200706,2005/0008254 and 2004/0130626 by Ouchi et al. describe an imageprocessing apparatus for generating graphics data representing a singleseamless planar image synthesized from a multiple sets of graphics datacontained in a plurality of graphics files which establishes aspheroidal or cylindrical projection plane for synthesis of the multiplesets of graphics data. The disclosure of each of these applications isfully incorporated herein by reference.

The following patents and applications are generally indicative ofcurrent panoramic technology in cameras “built in” to mobile phonehandsets. The list does not purport to be exhaustive.

International application WO/2005/041564 by KONINKLIJKE PHILIPSELECTRONICS N.V. describes a digital camera with panorama or mosaicfunctionality. The invention relates to an electronic device with adigital camera, and to a method of enabling to create a compositepicture using a digital camera. In some embodiments, the electronicdevice is a mobile phone. The disclosure of this application is fullyincorporated herein by reference.

United States Patent Application 20060181619 to Liow et al. describes amethod for forming a panoramic image using a camera. According to Liow,a blending area is formed in a first image on a display of the camera.The camera is moved before being prepared to take a second photograph.In the blending area only, a pixel matching process is performed fordetermining an alignment of the portion of the first image in theblending area and part of a second image of the second photograph in theblending area. In some embodiments, the method is practiced using acamera in a mobile phone. The disclosure of this application is fullyincorporated herein by reference.

There are commercially available panoramic picture technologiesavailable for mobile phones. These technologies include, but are notnecessarily limited to:

Scalado Autorama™ (Scalado AB; Sweden);

Vivid Panorama (Acrodea; Tokyo; Japan); and

PanoMan™ (Bit Side Technologies; Berlin; Germany).

Korean Patent 20060006186 to LG electronics describes taking a panoramicpicture while transparently showing the part of the panoramic picturealready taken.

SUMMARY OF THE INVENTION

An aspect of some embodiments of the invention relates to a userinterface for acquisition of frames of a panoramic image. In anexemplary embodiment of the invention, the frames are acquired insequential order.

In an exemplary embodiment of the invention, a border surrounds a firstacquired image frame in a viewfinder. Optionally, the user selects anedge or corner of the border to indicate a motion direction to acquirethe next image. Optionally, the user selects a motion direction byinitiating camera motion. In an exemplary embodiment of the invention,at least a portion of the selected edge or corner provides visual and/ordynamic feedback, e.g. by blinking or changing color as the user movesthe camera in the selected direction and/or when the next frameindicator is hidden. Optionally, the motion direction is predeterminedby the camera and/or user and a relevant edge (or portion thereof) isindicated. In an exemplary embodiment of the invention, the userindicates a motion direction by moving the camera and a relevant edge(or portion thereof) is indicated in the viewfinder.

In an exemplary embodiment of the invention, when a potential next frameis in the viewfinder, a feedback signal is provided. Optionally, thefeedback signal indicates to the user to stop moving the camera and/orindicates that a next frame has been automatically acquired by thecamera. Optionally, the feedback signal includes a change in a color ofthe border and/or audio signal (e.g. bell or voice prompt).

In another exemplary embodiment of the invention, a stationary aimingtarget is provided in the field of view of a first frame and a “nextframe” indicator is defined at a different position (e.g. at an edge) ofthe field of view when the first frame is acquired. According to thisexemplary embodiment of the invention, the user selects the next frameby moving the camera so that the next frame indicator progresses towards(optionally into) the aiming target. At this stage the next frame isacquired. In an exemplary embodiment of the invention, acquisition ofthe next frame is initiated automatically in response to positioning ofthe next frame indicator in the aiming target. Optionally, the aimingtarget is significantly smaller than the viewfinder screen. In anexemplary embodiment of the invention, the aiming target comprises anedge which is a sixteenth, optionally an eighth, optionally a quarter ofthe length of a corresponding edge of the viewfinder (or intermediaterelative lengths). In an exemplary embodiment of the invention, adecrease in size of the aiming target contributes to ease of aiming.

According to various exemplary embodiments of the invention, the aimingtarget is provided at a center of the viewfinder or at another location.

In an exemplary embodiment of the invention, a degree of mismatch isvisible in the viewfinder to the user throughout the aiming process.Optionally, a user interface evaluates a degree of zoom and/or distancein addition to a spatial alignment.

In an exemplary embodiment of the invention, the user plays a game aboutgetting the next frame indicator to the aiming target. Optionally thenext frame indicator is defined as a ball or portion of a ball.Optionally the user plays a game about getting the ball in the aimingtarget.

In an exemplary embodiment of the invention, the described userinterface contributes to improved quality of multidimensional panoramicimages (e.g. 3×3 or 4×4 frames) as opposed to conventional “strippanoramas” (e.g. 1×3 or 1×4). Optionally, improved quality ofmultidimensional panoramic images results from improved alignment offrames in multiple dimensions.

According to various exemplary embodiments of the invention, successiveframes in a panoramic image can be arranged horizontally (end-to-end),vertically (top-to-bottom) or diagonally (corner-to-corner).

An aspect of some embodiments of the invention relates to using cameramotion (CaMo) to automate selection of an image frame for constructionof a panoramic image within a mobile telephone platform. Optionally, thepanoramic image is of a large 3D surface. Optionally, sensing of CaMocontributes to automated guidance to a correct next frame. In anexemplary embodiment of the invention, a CaMo engine provides feedbackbased on sensed camera motion to a user about one or more of direction,panning speed, rotation angle and zoom. In an exemplary embodiment ofthe invention, the CaMo engine contributes to automatic acquisition of anext frame at a correct speed.

In an exemplary embodiment of the invention, a mobile telephone camerais equipped with a motion sensor that detects a motion direction and/orextent. Optionally, the mobile telephone camera provides feedbackregarding motion speed and/or motion direction and/or image suitabilityto a user. Optionally, presence of moving objects in a panoramic frameand/or presence of faces on an edge contributes to a degree ofunsuitability.

In an exemplary embodiment of the invention, mobile telephone camerasenses an appropriate “next frame” as the user moves the camera. In anexemplary embodiment of the invention, the “next frame” is acquired whenan acceptably small, optionally no, CaMo is detected. Optionally, thenext frame is acquired without the user pressing a shutter button.Optionally, the camera indicates to the user to press a shutter buttonto acquire a next frame. Optionally, a user acquires a next frame bypressing a shutter button without instruction from the camera.

An aspect of some embodiments of the invention relates to combiningvideo and still images to create a panoramic image within a mobiletelephone platform. In an exemplary embodiment of the invention, asingle frame selected from a video stream as a still image serves as aninitial frame for construction of a panoramic image. Optionally,additional frames selected from the video stream are stitched to theinitial frame to form the panoramic image. Optionally, one or more stillimages is captured and stitched to the initial frame to form thepanoramic image. In general, resolution of still photos tends to behigher than that achieved in video.

An aspect of some exemplary embodiments relates to interruptingacquisition of frames for a panoramic image and resuming acquisition ata later time. Optionally, interruption occurs to photograph a subjectnot related to the panorama. Optionally, interruption occurs because aninterfering object passes through the panoramic field of view. In anexemplary embodiment of the invention, a user interface featuringoverlay alignment contributes to an ability to resume panoramicacquisition. In an exemplary embodiment of the invention, a panoramicimage can be assembled by stitching together frames acquired out ofsequence based upon CaMo data for each of the frames.

According to some embodiments of the invention, if the CaMo engineand/or the user feel that alignment between successive frames is close,a new frame is captured. Optionally, this “pre-alignment” stepcontributes to increased ease in computerized stitching to form thepanoramic image.

In an exemplary embodiment of the invention, an image frame notspatially related to a sequence of panoramic image frames is acquiredand blended into the panoramic image. Optionally, the frame notspatially related to the sequence of panoramic image frames is acquiredbetween successive frames of the panoramic image.

In an exemplary embodiment of the invention, a user interface offers apossibility of continuing from a last acquired frame in the panorama byacquiring an overlapping next frame. Optionally, a degree of overlap is30, 40, 50 or 60%. Optionally, a 30 to 60% overlap contributes to anability of the user to correctly align the frames. In an exemplaryembodiment of the invention, the overlapping frames are displayed in anoverlay alignment with one frame moved “behind” the other frame.Optionally, overlay alignment contributes to an ability of the user toaim the camera correctly.

Optionally, a front frame of the overlay is semi transparent oralternating pixels from the two overlay frames are visible.

Optionally, a frame containing interference (e.g. a pedestrian orvehicle passing through a foreground of the frame) is deleted andrecaptured to complete the panoramic sequence of frames.

Optionally, a user standing at a first distance from a photographicsubject and acquiring sequential frames of a panoramic image moves to asecond distance from the subject, then returns to the first distance andresumes acquisition of sequential frames. Optionally the user acquiresunrelated frames at the second distance.

Optionally, acquisition of frames in a panoramic image is interrupted byan external stimulus. In an exemplary embodiment of the invention, thecamera is installed in a mobile phone handset and the external stimuluscomprises an incoming telephone call or message.

In an exemplary embodiment of the invention, 70 or 80 or 90% or greateridentity of pixels in an overlap area between adjacent framescontributes to a capacity to resume capture of an interrupted sequenceof panoramic image frames. Optionally, an increase in pixel identitycontributes to ease of matching when searching for a next frame toresume capture of a panoramic sequence of frames.

Optionally, unrelated or non-sequential or alternate range frames areblended with a sequence of frames forming a panoramic image. Optionally,the camera and/or the user select appropriate blending strategies.

An aspect of some embodiments of the invention relates to a 3Dprojection of a panoramic image within a mobile telephone platform.According to exemplary embodiments of the invention, a projection angleof the 3D projection can be variable from the mobile phone or fixed.

An aspect of some embodiments of the invention relates to inclusion ofvisually unrelated images in a panoramic image to form a collage.Optionally, the visually unrelated images comprise a sticker and/or atleast a portion of a spatially discontiguous frame.

An aspect of some embodiments of the invention relates to use of mosaictemplates in construction of a panoramic image.

Optionally, collage formation and/or template application are performedwithin a mobile phone handset.

In an exemplary embodiment of the invention, there is provided a methodof assembling a panoramic image from a plurality of frames in a camera,the method comprising:

-   (a) acquiring a video stream using a camera;-   (b) selecting an initial frame from the video stream for capture as    a still image;-   (c) further selecting at least one additional frame; and-   (d) stitching the at least one additional frame to the initial frame    in the camera to create a panoramic image.

Optionally, the camera is installed in a mobile phone handset.

Optionally, the additional frames are selected from the video stream.

Optionally, the at least one additional frame is captured as anadditional still image.

Optionally, the further selecting is performed via overlay alignment.

Optionally, the selection of frames is performed via overlay alignment.

Optionally, detection of camera motion (CaMo) contributes to the furtherselecting.

Optionally, detection of camera motion comprises detecting CaModirection.

Optionally, detection of CaMo indicates user's hand stability using CaMo(e.g. STRI's AutoShot functionality).

Optionally, detection of camera motion comprises detecting CaMo speed.

Optionally, detection of camera motion comprises providing feedback onCaMo.

Optionally, detection of camera motion comprises providing feedback onCaMo direction and/or CaMo speed.

Optionally, an absence of CaMo triggers acquisition of an additionalframe.

Optionally, selecting is performed manually by a user.

Optionally, the method comprises detection of at least one undesirablefeature.

Optionally, the undesirable feature comprises a face.

Optionally, the further selecting is performed automatically by thecamera.

Optionally, the further selecting is performed manually by a user.

Optionally, the panoramic image is of a 3D surface.

In an exemplary embodiment of the invention, there is provided a methodof providing a virtual 3D projection of a panoramic image in a camera,the method comprising:

-   (a) storing a panoramic image in a memory of a mobile telephone; and-   (b) applying a projection algorithm to produce a virtual 3D    projection of the panoramic image on a display of the camera.

Optionally, the camera is installed in a mobile phone handset.

Optionally, the method comprises:

(c) selecting a projection algorithm so that the 3D projection emulatesprojection on a surface with desired characteristics.

Optionally, the method comprises:

is performed via overlay alignment.

selecting a projection method that emulates a live video of a real-world3D scene by dynamically updating the device's screen according to achanging point of view.

Optionally, the method comprises:

dynamically updating the device's screen according to a changing pointof view to emulate a live video of a real-world 3D scene.

Optionally, the method comprises:

(c) selecting a projection algorithm so that the 3D projection emulatesprojection on a surface from a desired angle.

Optionally, the projection algorithm allows the user to dynamicallychange a projection angle to simulate a real-world look and feel.

In an exemplary embodiment of the invention, there is provided a userinterface for a mobile phone, the user interface adapted to aid the userto align a next frame of a panoramic image with a view area of a cameraincorporated in the phone, the interface comprising:

-   (a) a motion direction indicator visible to a user; and-   (b) a next frame alignment indication.

Optionally, the interface comprises a direction selection module adaptedfor selection of a motion direction for acquisition of the next frame.

Optionally, the visible motion direction indicator comprises at least aportion of an edge of the view area.

Optionally, the visible motion direction indicator comprises an arrow.

Optionally, a correct alignment of the next frame with the view areaproduces a stop signal which alerts the user to hold the camera inposition so that the next frame can be acquired.

Optionally, the next frame alignment indication comprises an aimingobject and an aiming target; wherein alignment of the aiming object withthe aiming target indicates that the next frame is properly aligned.

Optionally, the interface is organized as a game in which scoring isbased upon speed of alignment of the aiming object with the aimingtarget.

Optionally, the interface is used to create a matrix of frames which ispanoramic in two different dimensions.

In an exemplary embodiment of the invention, there is provided a methodof assembling a panoramic image using a camera, the method comprising:

-   (a) acquiring at least one frame belonging to a panoramic image    using the camera;-   (b) acquiring at least one frame not belonging to the panoramic    image;-   (c) resuming acquisition of frames belonging to a panoramic image;    and-   (d) assembling the panoramic image from at least one frame acquired    in (a) and at least one frame acquired in (c).

In an exemplary embodiment of the invention, there is provided a methodof assembling a panoramic image using a camera, the method comprising:

-   (a) acquiring at least one frame belonging to a panoramic image    using the camera;-   (b) interrupting the acquiring and changing an aiming direction of    the camera;-   (c) resuming acquisition of frames belonging to a panoramic image;    and-   (d) assembling the panoramic image from at least one frame acquired    in (a) and at least one frame acquired in (c).

Optionally, the camera is installed in a mobile phone handset.

Optionally, the at least one frame not belonging to the panoramic imageis excluded because it contains an interfering object.

Optionally, the at least one frame not belonging to the panoramic imageis acquired from a different position.

In an exemplary embodiment of the invention, there is provided a methodof producing a composite image on camera; the method comprising:

-   (a) providing a first image on a display of the camera;-   (b) selecting a second image;-   (c) indicating a position on the first image; and-   (d) applying the second image to the first image at the indicated    position to produce a composite image.

Optionally, the second image is smaller than the first image.

Optionally, the camera is installed in a mobile phone handset.

Optionally, the first image is a panoramic image.

Optionally, the second image comprises a clip-art graphic image.

Optionally, the second image comprises at least a portion of aphotographic image.

Optionally, the second image is acquired between two frames of a firstimage comprising a photographic panorama.

Optionally, the method comprises:

(e) assigning a degree of transparency to the second smaller image.

In an exemplary embodiment of the invention, there is provided a cameraadapted to assemble a panoramic image from a plurality of frames, thecamera comprising:

-   (a) a video acquisition module capable of providing a video stream;-   (b) a selector module adapted to select an initial frame from the    video stream for capture as a still image;-   (c) a user interface adapted to guide a user in selecting additional    frames; and-   (d) a stitching module adapted to stitch the additional frames to    the initial frame to create a panoramic image.

Optionally, the camera is provided as part of a mobile phone handset.

In an exemplary embodiment of the invention, there is provided a cameraadapted to provide a virtual 3D projection of a panoramic image, thecamera comprising:

-   (a) a memory adapted for storage therein of a panoramic image; and-   (b) analytic circuitry adapted to apply a projection algorithm to    produce a virtual 3D projection of the panoramic image on a display    of the camera.

Optionally, the camera is provided as part of a mobile phone handset.

In an exemplary embodiment of the invention, there is provided a mobilephone handset adapted to aid the user to align a next frame of apanoramic image with a view area of a camera incorporated in thehandset, the handset comprising circuitry adapted for presentation of auser interface comprising

-   (a) a visible motion direction indicator; and-   (b) a next frame alignment indication.

In an exemplary embodiment of the invention, there is provided a cameracomprising circuitry adapted to assemble a panoramic image, the cameracomprising:

-   (a) a frame acquisition module;-   (b) circuitry comprising a panoramic assembly module which permits    the user to:

(i) acquire at least one frame belonging to a panoramic image;

(ii) acquire at least one frame not belonging to the panoramic image;and

(iii) resume acquisition of frames belonging to the panoramic image; and

-   (d) a stitching engine adapted to assembling the panoramic image    from at least one frame acquired in (i) and at least one frame    acquired in (ii).

Optionally, the camera is provided as part of a mobile phone handset.

In an exemplary embodiment of the invention, there is provided a cameracomprising:

-   (a) a display adapted to present a first image;-   (b) a user accessible library of second images;-   (c) a selection module adapted for selection of a second image; and-   (d) a user operable alignment module adapted to apply the second    image to the first image at a selected location.

Optionally, the camera is provided as part of a mobile phone handset.

Optionally, some embodiments of the invention employ a remote server tomake decisions. Optionally, the decisions include one or more ofdecisions about quality, alignment, order and consistency of frames.

In an exemplary embodiment of the invention, there is provided a displayscreen of a mobile phone handset adapted to aid the user to align a nextframe of a panoramic image with a view area of a camera incorporated inthe handset, the handset comprising circuitry adapted for onscreenpresentation of:

-   (a) a motion direction indicator; and-   (b) a next frame alignment indication responsive to camera motion.

In an exemplary embodiment of the invention, there is provided a cameracomprising circuitry adapted to assemble a panoramic image, the cameracomprising:

-   (a) a frame acquisition module for acquiring frames;-   (b) a memory with plurality of frames stored therein, only some of    the frames belonging to a panoramic image;-   (c) a stitching engine adapted to identify those frames in the    memory which belong to the panoramic image and assemble the    identified frames in sequence.

Optionally, the camera is provided as part of a mobile phone handset.

In an exemplary embodiment of the invention, there is provided a cameracomprising:

-   (a) a display adapted to present a first image;-   (b) a user accessible library of second images;-   (c) a selection module adapted for selection of a second image; and-   (d) a user operable alignment module adapted to apply the second    image to the first image at a selected location.

Optionally, the camera is provided as part of a mobile phone handset.

In an exemplary embodiment of the invention, there is provided a cameracomprising:

-   (a) a user interface adapted for manual alignment of two frames via    overlay to produce a pair of pre-aligned frames; and-   (b) a stitching module adapted to receive the pre-aligned frames and    perform an exact alignment.

Optionally, the camera is provided as part of a mobile phone handset.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary non-limiting embodiments of the invention described in thefollowing description, read with reference to the figures attachedhereto. In the figures, identical and similar structures, elements orparts thereof that appear in more than one figure are generally labeledwith the same or similar references in the figures in which they appear.Dimensions of components and features shown in the figures are chosenprimarily for convenience and clarity of presentation and are notnecessarily to scale. The attached figures are:

FIGS. 1 a and 1 b are schematic representations of front and back sidesrespectively of a mobile telephone handset adapted to provide apanoramic imaging interface according to an exemplary embodiment of theinvention;

FIGS. 2 a; 2 b; 2 c and 2 d are flow diagrams illustrating exemplarymethods of panoramic picture acquisition and manipulation according tovarious exemplary embodiments of the invention;

FIGS. 3 a, 3 b, 3 c and 3 d are schematic representations of a mobilephone user interface for acquisition of frames of a panoramic imageaccording to exemplary embodiments of the invention;

FIG. 4 is a schematic representation of application of a graphic“sticker” to an image according to an exemplary embodiment of theinvention;

FIG. 5 is a schematic representation of blending of an additional imagefrom a series of stored images to an image according to an exemplaryembodiment of the invention;

FIG. 6 is a schematic representation of inversion of an image accordingto an exemplary embodiment of the invention;

FIG. 7 is a schematic representation of a panoramic mosaic templateaccording to an exemplary embodiment of the invention;

FIG. 8 is a schematic representation of an exemplary virtual 3Dprojection interface and exemplary 3D projection outputs; and

FIGS. 9 a; 9 b; 9 c and 9 d represent schematically different exemplarysequences for acquisition of frames in an exemplary panoramic matrix.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Overview

FIGS. 1 a and 1 b are schematic representations of front and back sidesrespectively of a mobile telephone handset adapted to provide apanoramic imaging interface according to one or more exemplaryembodiments of the invention. Exemplary handset 100 comprises a displayscreen 116 (FIG. 1 a), a standard twelve key telephone keypad 112 and anavigation device 110. Optionally one or more auxiliary function keys114 are included. In an exemplary embodiment of the invention, auxiliaryfunction keys 114 are used to control camera functions. In the depictedembodiment (FIG. 1 b) a lens 150 is visible on the back of handset 100.

In an exemplary embodiment of the invention, a memory device 120 storesa panoramic imaging module 190. Optionally, imaging module 190 isfurther subdivided into functional components. In the depictedembodiment, imaging module 190 comprises a camera interface 160, acamera motion (CaMo) engine 164, a stitching engine 162, and a panoramicuser interface 170.

In an exemplary embodiment of the invention, camera interface 160provides menus and/or instructions to a user on display 116. Optionally,camera interface 160 operates concurrently or sequentially withpresentation of a view area of lens 150 on display 116.

In an exemplary embodiment of the invention, CaMo engine 164 monitors adirection and/or velocity of camera motion as the user sweeps or pansacross an area comprising multiple fields of view of lens 150.Optionally, panning is horizontal, vertical or diagonal.

In an exemplary embodiment of the invention, panoramic user interface170 aids the user in adjusting panning direction and speed of motion forpanorama acquisition and/or selecting frames belonging to a panoramicimage and/or in manipulation of the panoramic image after it isacquired.

In an exemplary embodiment of the invention, stitching engine 162assembles or “stitches” the frames together to form the panoramic image.Optionally, a degree of overlap between successive frames of the imagecontributes to image clarity for static subjects (e.g. landscapes).Optionally, a degree of overlap between successive frames of the imagedetracts from image clarity for dynamic subjects (e.g. streets withmoving vehicles or crowds of people).

In an exemplary embodiment of the invention, the panoramic image is of a3D object. Optionally, different portions of the 3D object arephotographed from different angles.

Exemplary Methods

FIGS. 2 a; 2 b; 2 c and 2 d are flow diagrams illustrating exemplarymethods of panoramic picture acquisition and manipulation according tovarious exemplary embodiments of the invention.

FIG. 2 a is a simplified flow diagram depicting an exemplary method 200of acquisition of a panoramic image on a mobile phone handset (e.g.100). At 210 a user is presented with a menu and selects photography andpanorama mode within photography. At this stage lens 150 is activatedand display screen 116 begins to function as a viewfinder. The usercompares what is visible in viewfinder 116 to the entire scene beforethem and selects a frame to serve as a first frame for a panoramicseries. Optionally, this first frame comprises one end of a conventionalstrip panorama. Alternatively, or additionally, the first framecomprises one frame of a matrix which is panoramic in two dimensions. Inan exemplary embodiment of the invention, the camera is acquiring avideo stream when the first frame is acquired. Optionally, the cameracontinues to acquire a video stream as additional frames are acquired.

At 220, the selected frame is acquired and at 230 it is saved.

At 240, the camera asks the question: “Is another frame needed?”

In some exemplary embodiments of the invention, the user defines anumber of frames to be acquired ahead of time. According to theseexemplary embodiments of the invention, the camera compares a number offrames currently saved to the defined number of frames and ascertainsthe answer automatically.

In some exemplary embodiments of the invention, the user defines anumber of frames to be acquired during image frame acquisition.Definition of a number of frames can be accomplished, for example, bydefining an overall size of the panorama, for example by sweeping orpanning with the camera. According to these exemplary embodiments of theinvention, the user answers the question manually. Optionally, theanswer is conveyed to camera interface 160 by pressing one of auxiliarykeys 114. Optionally, a default number of frames (e.g. 3, 5 or 8) isselected automatically if no user input is received. Optionally, theanswer yes is assumed by default, and the user can override and answerno (meaning end of acquisition) by pressing one of auxiliary keys 114.

If the answer to the question is yes, acquisition 220 and saving 230 arerepeated.

When a sufficient number of frames have been acquired, the answer to thequestion is no. At this stage, all stored frames are assembled 250 toproduce a panoramic image. Optionally, assembly is performed bystitching engine 162.

In an exemplary embodiment of the invention, the panoramic image ispresented as a preview 260 on display 116. Optionally, the preview isnot subject to manipulation or editing at this stage. Optionally, theuser is asked if the previewed panorama should be stored for futuredisplay and/or editing and/or manipulation and/or transmission.Optionally, panoramic images produced by exemplary embodiments of theinvention can be transmitted to another device. Alternatively, oradditionally, the user can select to improve the image by indicating anarea to be reacquired. Optionally, the user interface guides the user toaim camera so as to reacquire the selected area.

In an exemplary embodiment of the invention, the other device comprisesone or more additional mobile phone handsets and the panoramic image isincluded in a Multi-Media Message Service (MMS) message. Alternatively,or additionally, transfer to another mobile phone handset can beperformed locally, e.g. via Bluetooth, infrared or another wirelessprotocol.

In an exemplary embodiment of the invention, the other device comprisesone or more remote servers and the panoramic image is up-loaded to theserver (e.g. using WAP).

FIG. 2 b is a simplified flow diagram depicting an exemplary method 242which aids a user in selecting frames for inclusion in the panoramicimage by means of a user interface in greater detail.

At 240, the camera asks the question: “Is another frame needed?” (as inFIG. 2 a).

In response to a “yes” answer which can be delivered automatically ormanually as described above, the user is presented with an indication243 in which direction to move the camera of handset 100.

At 244, CaMo is sensed. In an exemplary embodiment of the invention,sensing is performed by CaMo engine 164.

At 246, an indication of an amount of progress towards a next frame isindicated graphically to the user. In an exemplary embodiment of theinvention, graphic indications are presented in display 116 over acurrently visible viewing area of lens 150.

At 248, a notification to the user to stop to acquire the next frame isprovided. Optionally, the next frame is acquired (220; FIG. 2 a)manually or automatically as described hereinabove.

In an exemplary embodiment of the invention, panoramic user interface170 (FIG. 1 a) comprises panoramic editing interface 270 which isdescribed in greater detail in FIG. 2 c.

FIG. 2 c is a simplified flow diagram depicting exemplary methods 262 ofediting and/or manipulation of an acquired panoramic image in a mobiletelephone handset according to exemplary embodiments of the invention.

The flow diagram begins with preview panorama (260 of FIG. 2 a)indicating that a complete panoramic image has been acquired and isavailable.

With a previewed panorama 260 visible on display 116 (FIG. 1), the userselects a panoramic editing interface 270. Optionally, editing interface270 offers the user more than one editing possibility. In the depictedembodiment, three main editing possibilities are offered, although theactual number of options may be greater than three or less than threeaccording to different exemplary embodiments of the invention.Optionally, there is a trade off between number of editing possibilitiesoffered and available hardware memory and/or computing capacity inmobile phone handset 100.

The three main editing possibilities depicted in FIG. 2 c areapplication of a sticker to a panoramic image, blending of an image to apanorama and application of a mosaic template to a panorama. Accordingto exemplary embodiments of the invention, the user selects one or moreof these possibilities.

In an exemplary embodiment of the invention, the user applies a stickerto a panoramic image.

At 280 a user selects a sticker. The stickers are optionally provided asclip-art in any available image format including, but not limited to,jpeg, tiff and gif. Typically, there is a trade off between the numberof stickers available for selection and available hardware memory inmobile phone handset 100. Optionally, availability of stickers fordownload from a remote server contributes to a reduction in demands onavailable hardware memory in mobile phone handset 100.

At 282, the user selects a location within the panoramic image.Optionally, location selection is by manipulation of a cursor on display116 using navigation device 110. In an exemplary embodiment of theinvention, automating sensing of CaMo is used for cursor manipulation.Optionally, the user moves the cursor by moving the camera.

At 284 the user applies the sticker to the selected location.Optionally, application is by operation of an auxiliary key 114 or byselection of an “apply” command from a menu. Optionally, stickerselection 280 and application 284 are integrated into a singleoperation. Optionally, application 284 is performed prior to locationselection 282 and the selected sticker serves as a cursor.

In an exemplary embodiment of the invention, the user blends aphotographic image to a panoramic image.

At 286, the user selects an image to blend into the panorama. Again,there is a trade off between the number of images available forselection and available hardware memory in mobile phone handset 100.Optionally, availability of images for blending as downloadable itemsfrom a remote server contributes to a reduction in demands on availablehardware memory in mobile phone handset 100.

At 288 the user selects a location within the panoramic image.Optionally, location selection is as described above for stickerapplication.

At 290 the user blends the selected image to the selected location.Optionally, blending is as described above for sticker application.Optionally, blending 290 and image selection 286 are performedconcurrently. Optionally, an order of the operations is changed so thatselecting 288 of the blending location includes moving the imageselected at 288 over the panorama. Optionally, a portion of the selectedimage is moved over the panoramic image as an overlay. In an exemplaryembodiment of the invention, moving at least a portion of the image overthe panoramic image contributes to an ability to achieve a correctalignment. Optionally, CaMo and/or a navigation device are employed tomove the overlaid image with respect to the panorama. In an exemplaryembodiment of the invention, analytic circuitry provides a precisealignment based upon an initial alignment provided by the user.

In an exemplary embodiment of the invention, the user applies a mosaictemplate (e.g. template 700 of FIG. 7; described hereinbelow) to apanoramic image. Optionally, application of a template to an assembledpanoramic image contributes to a reduction in perceptibility ofimperfections at stitching junctions and/or to a reduction inperceptibility of parametric differences between frames. Parametricdifferences between frames include, but are not limited to, whitebalance, lighting level, contrast, viewing angle and applied degree ofzoom.

At 292, the user selects a mosaic template from among a group of mosaictemplates. Again, there is a trade off between the number of templatesavailable for selection and available hardware memory in mobile phonehandset 100.

At 294, the user optionally adds text to the template. At 296, the userapplies the selected template to the panoramic image to create themosaic.

FIG. 2 d is a simplified flow diagram depicting a method 272 for virtualprojection of a panoramic image on a 3D surface.

In an exemplary embodiment of the invention, method 272 begins afteracquisition of a panoramic image and preview 260 (FIG. 2 a) of thepanorama on display 116 (FIG. 1). In other exemplary embodiments of theinvention, method 272 begins whenever the user wishes to view a recordedpanorama.

At 274 the user selects a 3D projection interface 274. Optionally,projection interface 274 is provided as a part of panoramic userinterface 170.

At 276, the user optionally selects a projection surface. Exemplaryprojection surfaces include, but are not limited to, a concave surface,a convex surface and a sphere. In an exemplary embodiment of theinvention, selection of a surface indicates to panoramic user interface170 which projection algorithm to apply. 3D projection algorithms areknown to those of ordinary skill in the art. Algorithms suitable for usein the context of the present invention can be found, for example, in“Image Alignment and Stitching: A Tutorial” by Richard Szeliski(Preliminary draft, Sep. 27, 2004, Technical Report, MSR-TR-2004-92.pages 8-10; which is fully incorporated herein by reference) and/or inU.S. patent applications 2006/0062487, 2006/0018547, 2005/0200706,2005/0008254 and 2004/0130626 by Ouchi et al. which are each fullyincorporated herein by reference. Additional algorithms suitable for usein the context of the present invention are disclosed in a relatedapplication Ser. No. 11/826,772 entitled “METHOD FOR CONSTRUCTING ACOMPOSITE IMAGE” by Noam SOREK and Orna BREGMAN AMITAI filed the samedate as the instant application and fully incorporated herein byreference.

At 278, virtual projection of the panorama on the 3D surface isperformed and the result is displayed on display 116 and/or stored in amemory. Optionally, a stored virtual projection can be transmitted toanother device (e.g. another mobile phone handset) and/or viewed in thefuture. Optionally, virtual projection 278 occurs automatically afterselection 276 of the projection surface. In an exemplary embodiment ofthe invention, only one projection surface is available and selection of3D projection interface 274 leads directly to virtual projection 278.

Optionally, (279) the user can vary the projection angle and zoom withrespect to the selected surface. Optionally, this variation can beperformed before or during virtual projection 278. In an exemplaryembodiment of the invention, the user moves the camera and the CaMoengine interprets the motion and alters the apparent projection angleaccordingly. Mobile phone handsets 100 with greater memory and/orcomputing capacity will typically be able to more rapidly adjust virtualprojection 278 in response to variations in projection angle 279specified by the user.

Exemplary Next Frame Alignment Interface

FIGS. 3 a, 3 b, 3 c and 3 d are schematic representations of a mobilephone user interface for acquisition of frames of a panoramic imageaccording to exemplary embodiments of the invention. These figuresillustrate one or more exemplary modes of practice of method 242 (FIG. 2b).

FIG. 3 a depicts a series of scenes (from left to right) visible in aviewing frame 310 on a display screen 116 of mobile phone handset (100of FIG. 1 a) as the user acquires a first image frame of a panoramicimage and moves the camera to the right in order to line up a next frameof the panoramic image. The series of scenes illustrate optionalfeatures of an exemplary user interface 300 according to an exemplaryembodiment of the invention.

In the left most panel, the user selects a first frame including twoobjects (tree 302 and mountain 304). A center target 320 issubstantially empty in this view. The user initiates panoramicacquisition, for example by pressing a shutter button (e.g. one ofauxiliary keys 114). In an exemplary embodiment of the invention, theuser then selects a panoramic orientation by beginning to move thecamera. In the depicted example, camera motion (CaMo) is to the right.

Automatic sensing of CaMo is well known to those of ordinary skill inthe art and is described, for example, in co-pending U.S. patentapplication 20070041058 entitled “A Device and A Method For IdentifyingMovement Patterns” filed on Aug. 14, 2006 by Eli BEN-AMI and IsraelDISATNIK and 20070041616 entitled “A Displacement and Tilt DetectionMethod For A Portable Autonomous Device Having An Integrated ImageSensor and A Device Therefor” filed on Aug. 14, 2006 by Jonggoo LEE, EliBEN-AMI, Israel DISATNIK and Natan LINDER. The disclosures of theseapplications are each incorporated herein by reference.

In an exemplary embodiment of the invention, combining “featuredetection” capabilities with the CaMo engine can affect a desired degreeof overlap. For example, if faces are detected, the overlap area may beenlarged to ensure that a complete face will be taken from one pictureand will not be part of the stitching area. Optionally, it isadvantageous to make this determination prior to looking for a nextframe. According to different exemplary embodiment of the invention,this capability can be optional (i.e. subject to user selection) orintegrated. A level of importance of this feature optionally varies witha degree of acquaintance of the photographer to subjects of thephotograph. Optionally, the user indicates whether a particular faceshould, or should not, be allowed to include a stitch line. In anotherexample, if faces are detected, the camera can suggest taking picturesso that faces are overlapped.

In the second panel from the left, CaMo engine 164 has sensed the cameramotion and provided a direction indicator 330. Optionally, directionindicator 330 blinks. Although direction indicator 330 is depicted as arectangle, it can be provided as any desired shape. Optionally,direction indicator 330 is provided as an arrow. Optionally, directionindicator 330 can occupy an entire edge of viewing frame 310. In anexemplary embodiment of the invention, direction indicator 330 helps theuser to move the camera properly. Optionally, direction indicator 330can also indicate speed of CaMo. For example, if speed of CaMo is toohigh, indicator 330 may turn yellow, then orange and eventually red. Ifspeed of CaMo is within an acceptable range, indicator 330 may turngreen. Other feedback means including, but not limited to, color and/orgraphic and/or iconic feedback and/or audio feedback (e.g. speech and/orsounds) can be employed in this context.

As the camera pans to the right, tree 302 disappears out of the leftside of viewing frame 310 and mountain 304 occupies progressively moreof frame 310.

The third panel from the left depicts two additional exemplary featuresof user interface 300.

In an exemplary embodiment of the invention, a “next frame border” 350is provided on display 116. Optionally, next frame border 350 can aidthe user in vertical and/or horizontal alignment of the next frame witha previous frame of the panoramic image under construction. As the userpans to the right, next frame border 350 covers progressively more ofviewing frame 310. If the user moves the camera up or down, next frameborder 350 becomes vertically misaligned with viewing frame 310 and theuser intuitively understands what correction to CaMo is desirable toachieve proper alignment.

Alternatively, or additionally, a “next frame center indicator” 340 isprovided. Next frame center indicator 340 is to be aligned with centertarget 320 for acquisition of the next frame. As with next frame border350, the user intuitively adjusts CaMo to align center indicator 340with center target 320.

The right most panel depicts correct alignment of the next frame withviewing frame 310. At this stage next frame border 350 is fully alignedwith viewing frame 310 and/or next frame center indicator 340 is fullyaligned with center target 320. In an exemplary embodiment of theinvention, CaMo is briefly interrupted at this to permit capture of thenext image frame. Optionally, capture of the next image frame can beautomatic or manual. In an exemplary embodiment of the invention,capture of the next image frame is automatic and user interface 300provides a signal to the user to cease CaMo. According to exemplaryembodiments of the invention, the signal can include one or more of achange in appearance of next frame border 350, a change in appearance ofnext frame center indicator 340, a change in appearance of directionindicator 330 and, an audible signal (e.g. tone, bell or synthetic voiceprompt). Change of appearance as used here can indicate one or more of achange in color, texture, degree of transparency and blinking rate.

In some exemplary embodiments of interface 300, both next frame border350 and next frame center indicator 340 are provided. In other exemplaryembodiments of the invention, only one of next frame border 350 and nextframe center indicator 340 is provided.

Optionally, alignment of the next frame is presented to the user as agame. In an exemplary embodiment of the invention, center target 320 andcenter indicator 340 are graphically modified to conform to a theme.Optionally the theme is a sports theme. For example, center target 320can be presented as a goal and center indicator 340 can be presented asa football. Optionally, a time used to place the ball in the goal ispresented to the user.

In an exemplary embodiment of the invention, center target 320 and/orcenter indicator 340 are sized to occupy only a small portion of viewingframe 310. Optionally, a small size of center target 320 and/or centerindicator 340 contributes to an ability of the user to align theseobjects. Optionally, center target 320 and center indicator 340 are asame (e.g. two rectangles of different sizes) or different shapes (e.g.a circle sized to be inscribed in a square). In an exemplary embodimentof the invention, center indicator 340 is configured as a cross andcenter target 320 is a geometric shape (e.g. circle or square) in whichthe cross is to be inscribed. Optionally, inscription of a cross in ageometric shape is intuitively understood by the user.

In an exemplary embodiment of the invention, the target is provided in alocation other than a center of viewing frame 310. Optionally, aposition of the target is adjusted relative to a desired degree ofoverlap of consecutive frames.

FIG. 3D illustrates exemplary ways of indicating to the user that CaMois inappropriate for aligning the next frame. In the depicted exemplaryframe, mountain 304 is still in view, but the user has shifted thecamera upwards. The type and degree of misalignment are optionallypresented to the user in one or more of at least three exemplary ways.

In a first optional information presentation mode, motion directionindicator 330 moves from its nominal position in the center of the rightedge of viewing frame 310 to a lower right corner of viewing frame 310.This indicates to the user that the camera should be shifted down whilemotion to the right continues.

In a second optional information presentation mode, next frame centerindicator 340 appears below center target 320 and the user understandsintuitively that the camera should be lowered to bring 340 and 320 intovertical alignment with one another.

In a third optional information presentation mode, next frame border 350appears below viewing frame 310 and the user understands intuitivelythat the camera should be lowered to bring 350 and 310 into verticalalignment with one another.

In another optional information presentation mode (depicted in FIG. 3b), next frame center indicator 340 appears above center target 320 andthe user understands intuitively that the camera should be lowered tobring 340 and 320 into vertical alignment with one another.

In another optional information presentation mode (also depicted in FIG.3 b), next frame border 350 appears above viewing frame 310 and the userunderstands intuitively that the camera should be lowered to bring 350and 310 into vertical alignment with one another.

In an exemplary embodiment of the invention, two, optionally three ofthese information presentation modes are used in conjunction with oneanother as depicted in FIG. 3 d, although any one of the three can besufficient to cause the user to implement a proper corrective motion ofthe camera.

FIG. 3 c illustrates an exemplary way of indicating to the user thatCaMo is inappropriate for aligning the next frame. In the depictedexemplary frame a gross misalignment has occurred so that mountain 304is no longer visible because the user has moved the camera too far up.The user sees irrelevant objects (e.g., bird 304, cloud 306 and sun 308)in viewing frame 310 but has no intuitive understanding of a spatialrelationship between these objects and a desired next frame for thepanoramic image under construction. In the depicted embodiment, CaMoengine 164 aids the user by providing an arrow 358 indicative of adirection to move the camera so that next frame border 350 and/or nextframe center indicator 340 can be displayed in at least a portion ofviewing frame 310. In other embodiments, CaMo engine 164 aids the userby providing an arrow 358 indicative of a direction to move the image sothat next frame border 350 and/or next frame center indicator 340 can bedisplayed in at least a portion of viewing frame 310.

Arrow 358 is depicted in aiming target 320 but can optionally bepresented anywhere in viewing frame 310. Optionally, a size of arrow 358indicates an amount of motion to be undertaken for the desiredcorrection. In an exemplary embodiment of the invention, as the usermoves the camera in a correct direction, arrow 358 shrinks until iteventually disappears and is replaced by one or more of next frameborder 350, next frame center indicator 340 and motion directionindicator 330. Optionally, a situation similar to that depicted in FIG.3 b provides a transition between the gross misalignment of FIG. 3 c andproper alignment of FIG. 3 a.

In an exemplary embodiment of the invention, arrow 358 can point in oneof four directions (up, down, left and right). According to thisembodiment of the invention, interface 300 performs a diagonalcorrection in two separate steps (e.g. down and left).

In an exemplary embodiment of the invention, arrow 358 can point in oneof eight directions (up, down, left, right and four diagonaldirections). According to this embodiment of the invention, interface300 performs corrections in a single step.

Exemplary Collage Production

According to some exemplary embodiments of the invention, a panoramicimage is incorporated into a collage on a mobile phone handset.Optionally, the collage can include clip art and/or photographic images(or portions thereof) in addition to the panoramic image. FIGS. 4 and 5depict exemplary user interfaces 400 and 500 respectively for collageproduction.

FIG. 4 is a schematic representation of exemplary user interface 400 forapplication of a graphic clip-art “sticker” to an image according to anexemplary embodiment of the invention.

According to depicted exemplary interface 400, display 416 concurrentlypresents panoramic image 460, and a library 420 of clip art stickers. Inan exemplary embodiment of the invention, the user selects a clip artitem (e.g. heart 422) using navigation device 110. Referring now to thesecond panel of FIG. 4, the user positions a cursor 430 at a desiredlocation in image 460 using navigation device 110. Optionally, a currentdirection of motion of cursor 430 is indicated by direction indicator432. In an exemplary embodiment of the invention, the user issues an“apply sticker command” when cursor 430 is positioned at a desiredlocation in image 460. As depicted in the right hand panel, the selectedclip-art (e.g. heart 422) is applied to image 460 as a sticker inresponse to the command. Optionally, the command is issued using one ofauxiliary keys 114.

FIG. 5 is a schematic representation of exemplary user interface 500 forblending of an additional photographic image from a series of storedimages to a panoramic image according to an exemplary embodiment of theinvention.

According to depicted exemplary interface 500, display 516 concurrentlypresents panoramic image 560, and a library 520 of photographic images(e.g. 522, 524 and 526). In an exemplary embodiment of the invention,the user selects a photographic image (e.g. 524) using navigation device110. Optionally, a selection direction within library 520 is indicatedby direction indicator 528. Referring now to the second panel of FIG. 5,the user positions a cursor 530 at a desired location in image 560 usingnavigation device 110. In an exemplary embodiment of the invention,automating sensing of CaMo is used for cursor manipulation.

Optionally, a current direction of motion of cursor 530 is indicated bydirection indicator 532. In an exemplary embodiment of the invention,the user issues a “blend photo” command when cursor 530 is positioned ata desired location in image 560. As depicted in the right hand panel,selected photograph 524 is blended to image 560 in response to thecommand. Optionally, the command is issued using one of auxiliary keys114. According to exemplary embodiments of the invention, applied image524 can be opaque or partially transparent. Optionally, applied image524 is opaque at its center and becomes progressively more transparenttowards its edges. In an exemplary embodiment of the invention, theprogression towards transparency contributes to a blending effect ofapplied image 524 with background image 560.

Exemplary Overlay Alignment

In an exemplary embodiment of the invention, a user interface providesan overlaid portion of a second image for alignment with a portion of afirst image. In some embodiments, overlay alignment contributes to asmooth transition between frames of a panoramic image by helping theuser achieve a good initial alignment. In an exemplary embodiment of theinvention, a first image is presented in a viewing frame. Optionally,the first image is a current “viewfinder” image which has not yet beenacquired. Typically, the viewfinder image resides in a temporary storagebuffer, sometimes at a lower resolution than “acquired” frames stored inmemory. A second image (e.g. a last acquired frame in a panoramic image)can then be overlaid in a portion of the viewing frame. Optionally, theoverlaid image is blended with the current viewfinder image. In anexemplary embodiment of the invention, blending of the overlaid imagewith the current viewfinder image produces a new “second image” whichcan be overlaid on a portion of a viewfinder image as panning continues.In this way, the user can piece together the panorama during frameacquisition.

Optionally, the user selects a portion of the second image to beoverlaid. In an exemplary embodiment of the invention, 30-60% overlayaids the user in seeking a correct alignment. Optionally, an increase inoverlay percentage contributes to ease of user managed alignment. In anexemplary embodiment of the invention, the user manipulates the firstimage with respect to the second overlaid image by moving the camerauntil a desired alignment is achieved. Optionally, the user or thecamera determines when desired alignment is achieved, for example bypixel matching. Optionally, a first image comprising a viewfinder imageis automatically acquired when the desired alignment is achieved. In anexemplary embodiment of the invention, stitching engine 162 adjusts thedesired alignment of two images in memory automatically. In manyexemplary embodiments of the invention, automatic alignment by stitchingengine 162 supplements and/or completes a preliminary alignmentperformed by the user.

In some exemplary embodiments of the invention, a portion of theviewfinder displays a previously acquired portion of a panoramic imageand the user employs overlay alignment to indicate a desired spatialrelationship between the new frame and the existing portion panoramicimage.

In some exemplary embodiments of the invention, the user constructs apanoramic image using overlay alignment of frames that were notoriginally acquired in a panoramic mode, or belong to separatepanoramas.

Exemplary Image Inversion

FIG. 6 is a schematic representation of a user interface 600 forinversion of an image according to an exemplary embodiment of theinvention. In the depicted embodiment a panoramic image 660 is presentedon a display screen 616. In an exemplary embodiment of the invention,the user selects “invert” from a menu and a mirror image panorama 662 ispresented on display 616. While horizontal inversion is depicted,vertical inversion is also within the scope of the invention.Optionally, inversion can be performed to allow incorporation ofadditional frames acquired from a different angle.

Exemplary Panoramic Template

In an exemplary embodiment of the invention, a panoramic image ispresented in a template on a display of a mobile telephone handset.

FIG. 7 is a schematic representation of a template 700 according to anexemplary embodiment of the invention. In the depicted template 700, abackground area 702 contains placeholders 710, 720 and 730 for frames 1,2 and 3 of a panoramic image respectively. Optionally, the user placespreviously acquired frames into placeholders 710, 720 and 730 withoutstitching the frames together in a standard panorama. Optionally, anadditional frame in a different orientation from the panorama is placedin an additional placeholder 740.

In an exemplary embodiment of the invention, frame separators 712, 722and 732 are provided between the placeholders to create a mosaic effect.In an exemplary embodiment of the invention, the frame separators serveto at least partially mask an imperfect alignment between adjacentframes.

In an exemplary embodiment of the invention, template 700 includes atleast one text box 750 for optional text entry. Optionally, the user canformat text using fonts and/or styles (e.g. italics, underline or bold)and/or effects (e.g. shadows, 3D or animation effects).

In an exemplary embodiment of the invention, the user can select a colorand/or texture and/or fill effect (e.g. gradient, recessed and/orsuperimposed) and/or theme for background area 702. Optionally,separators 712, 722 and 732 are treated as part of the background areain terms of color and/or texture and/or fill effect.

Optionally, background 702 and/or separators 712, 722 and 732 arerecessed with respect to the panoramic image so that the image casts ashadow on the background and/or separators.

Optionally, background 702 and/or separators 712, 722 and 732 aresuperimposed with respect to the panoramic image so that the backgroundand/or separators cast a shadow on the image.

In an exemplary embodiment of the invention, a user assembles a seriesof frames which do not form a panorama into placeholders 710, 720 and730 (and optionally 740). Optionally, the frames are thematicallyrelated (e.g. all pictures from a single location) but are notperceptibly spatially aligned as frames in a conventional panoramicimage are. For Example, a visitor at an Alpine chalet might assembleframes comprising a view of a valley from the front porch, a view of afacade of the chalet and a self portrait with the valley in thebackground into placeholders 710, 720 and 730 to create a theme-o-ramaof a visit.

Exemplary Virtual 3D Projection

FIG. 8 is a schematic representation of an exemplary user interface 800for virtual 3D projection interface with exemplary 3D projection outputsof panoramic images depicted below.

The upper portion of FIG. 8 shows a panoramic image 860 on a displayscreen 816 of mobile handset 100. In order to emphasize the effect ofprojection on different types of 3D surfaces, image 860, includes thenumeral 1 through 5 in a uniform size arranged from left to right.Optionally, menu 820 presents a list of 3D projection surfaces. In thedepicted exemplary embodiment menu 820 contains three choices: convex(822), concave (824) and sphere (826). Selection of a 3D surface frommenu 820 can be via navigation device 110 as described above.

The three 3D surfaces depicted are exemplary only and projection on anysurface (e.g. face) or solid (e.g. cube) is possible. Optionally, alibrary of 3D surfaces for projection of panoramic images thereupon isavailable on a remote server and users can download different surfacesto their handsets for projection.

The lower portion of FIG. 8 shows exemplary 3D projection outputsresulting from selection of convex (822), concave (824) and sphere (826)respectively.

Selection of convex projection surface 822 from menu 820 producesmodified panoramic image 862. In image 862, a central portion(represented by the numeral 3) is modified relative to a correspondingarea of source image 860 using a relevant 3D projection formula. Endportions of modified image 862 (represented by the numerals 1 and 5) aremodified relative to a corresponding area of source image 860 accordingto the same projection algorithm. Intervening portions of image 862(represented by numerals 2 and 4) are less modified relative tocorresponding areas of source image 860. Optionally, the originalpanoramic image (before projection) and/or the projected image can bestored in memory of the handset and/or transmitted to another device(e.g. an additional mobile phone handset).

Selection of concave projection surface 824 from menu 820 producesmodified panoramic image 864. In image 864, a central portion(represented by the numeral 3) is modified relative to a correspondingarea of source image 860 using a relevant 3D projection algorithm. Endportions of modified image 864 (represented by the numerals 1 and 5) aremodified relative to a corresponding area of source image 860 in adifferent way according to the same algorithm. Intervening portions ofimage 864 (represented by numerals 2 and 4) are modified to a lesserdegree relative to corresponding areas of source image 860.

Selection of spherical projection surface 826 from menu 820 producesmodified panoramic image 866. In image 866, a central portion(represented by the numeral 3) is modified relative to a correspondingarea of source image 860 to a greater degree than in modified image 862using a relevant 3D projection formula. End portions of modified image866 (represented by the numerals 1 and 5) are modified relative to acorresponding area of source image 860 to a greater degree than inmodified image 862. In addition, modified image 866 is subject towidthwise compression with the greater compression occurring at the ends(represented by numerals 1 and 5).

In an exemplary embodiment of the invention, 3D projection contributesto a more realistic appearance of a panoramic image. Optionally, theuser can change the projection angle (e.g. by moving the camera) tosimulate turning their head in an actual scene. In an exemplaryembodiment of the invention, only a portion of the panoramic image ispresented on the display screen and the viewer can change the presentedportion by changing the selected projection angle. Optionally, changingthe presented portion by changing the selected projection anglesimulates a real-world feeling. Optionally, a degree of curvature and/orregularity of curvature are adjustable by the user. Optionally, the usercan zoom in and out of the projected image.

Exemplary Panoramic Matrices

In examples set forth hereinabove, the term “panoramic image” hasindicated a linear series of frames assembled end-to-end (orcorner-to-corner) to produce a panorama. In an exemplary embodiment ofthe invention, an image which is panoramic in two orthogonal dimensionsis produced by acquiring a series of frames arranged in a matrix. Anexemplary 3×3 matrix is presented here although larger matrices and/ormatrices which are larger in one orthogonal direction than in a secondorthogonal direction are within the scope of the invention. The numberof image frames in a matrix is typically influenced by interplay betweenimage resolution, available memory, available computing capacity andsensitivity of CaMo engine 164 or accelerometer. Alignment of a nextframe with a preceding frame is optionally according to user interfacesdescribed hereinabove.

Although frames are depicted as squares in the examples set forthhereinbelow, each frame can have any H:W aspect ratio. In an exemplaryembodiment of the invention, CaMo engine 164 is programmed to consideran aspect ratio of each frame. Optionally, aspect ratio and/or panningdirection are considered by CaMo engine 164 when deciding when toacquire a next frame.

FIGS. 9 a; 9 b; 9 c and 9 d represent schematically different exemplarysequences for acquisition of frames in an exemplary 3×3 panoramicmatrix.

FIG. 9 a depicts an exemplary acquisition sequence 900 in which the userbegins at the corner (top left in this example) of a desired image area.After acquiring frame 1, the user pans in a first direction (right inthis example) to acquire frame 2 and continues to pan to acquire frame3. After acquisition of frame 3, the user pans back (left in thisexample) until frame 1 is visible again, and then pans down to captureframe 4. The user then pans in the first direction again (right in thisexample) to capture frames 5 and 6. After acquisition of frame 6, theuser pans back (left in this example) until frame 4 is visible again,and then pans down to capture frame 7. The user then pans right tocapture frames 8 and 9 and the matrix is complete. Leftward panning fromframe 3 back to frame 1 and from frame 6 back to frame 4 can beperceived as a disadvantage by the user.

FIG. 9 b depicts an exemplary acquisition sequence 902 in which the userbegins at the center of a desired image area. After acquiring frame 1,the user pans in a first direction (right in this example) and acquiresframe 2. The user then pans in a direction perpendicular to the firstdirection (up in this example) and acquire frame 3. The user then pansin a direction opposite to the first direction (left in this example)and acquires frames 4 and 5. The user then pans in a direction oppositeto the second direction (down in this example) and acquires frames 6 and7. The user then resumes panning in the first direction and acquiresframes 8 and 9. This exemplary acquisition sequence eliminates panningback through previously acquired frames although it includes 4 changesin panning direction.

FIG. 9 c depicts an exemplary acquisition sequence 904 in which the userbegins at the corner (top left in this example) of a desired image area.After acquiring frame 1, the user pans in a first direction (right inthis example) to acquire frame 2 and continues to pan to acquire frame3. After acquisition of frame 3, the user pans in a perpendiculardirection (down in this example) to acquire frame 4 and then pans back(left in this example) to acquire frames 5 and 6. After acquisition offrame 6, the user pans in the perpendicular direction to acquire frame 7and resumes panning in the first direction to acquire frames 8 and 9.

FIG. 9 d depicts an exemplary acquisition sequence 900 in which the userbegins at the corner (top left in this example) of a desired image areaand acquires a last frame in the center of the image area. Afteracquiring frame 1, the user pans in a first direction (down in thisexample) to acquire frame 2 and continues to pan to acquire frame 3.After acquisition of frame 3, the user pans in a perpendicular direction(right in this example) to acquire frames 4 and 5 and then pans in anopposite direction (up in this example) to acquire frames 6 and 7. Afteracquisition of frame 7, the user pans back towards frame 1 (left in thisexample) to acquire frame 8 and resumes panning in the first directionto acquire frame 9.

In some exemplary embodiments of the invention, only a portion of amatrix is of interest. Referring again to FIG. 9 c, optionally a userbegins by acquiring frame 3, and then acquires frames 4, 5, 7, 8 and 9to produce an image that approximates a triangle and is panoramic in twoorthogonal directions. Alternatively (referring to FIG. 9 d) a userbegins by acquiring frame 1, then acquires frames 2, 3, 4 and 5 toproduce an “L” shaped image that is panoramic in two orthogonaldirections.

Exemplary Flat Projection of 3D Objects

In an exemplary embodiment of the invention, a flat panoramic image of a3D surface is provided. For example, two or three or four sides of abuilding can be depicted as frames in a single flat panoramic image.

General

In an exemplary embodiment of the invention, hardware sensors (e.g.accelerometers) at least partially replace the functionality of CaMoengine 164.

In an exemplary embodiment of the invention, user perceptible “front endfeatures” operate together with one or more “back end” features.Optionally, back end features are at least partially imperceptible,optionally substantially completely imperceptible to the user. In anexemplary embodiment of the invention, the user perceives improved imagequality resulting from back end features. Optionally, operation ofback-end features does not significantly adversely affect camera and/orhandset function.

One exemplary back end feature employed in some embodiments of theinvention comprises use of cumulative camera motion between two or moreframes of a panoramic image as a guide for stitching. Optionally,cumulative camera motion provides an initial alignment for stitching anda stitching algorithm makes fine adjustments in alignment based uponanalysis of pixels in proximity to the stitch line suggested byaccumulated CaMo.

Another exemplary back end feature employed in some embodiments of theinvention comprises automated image capture by the camera when the CaMoengine feels that a suitable next frame for a panoramic image is inview. Optionally, an initial capture of a next frame is at a lowresolution and/or includes some defect (e.g. motion, or blur). In anexemplary embodiment of the invention, if the user keeps the same nextframe in view, the camera automatically acquires one or more additionalimages. Optionally, when a panoramic image is being assembled, a “bestimage” for each frame is selected for inclusion in the panorama.According to various exemplary embodiments of the invention, selectionof a best image is made based on one or more parametric measures ofimage quality. Parametric measures of image quality include, but are notlimited to measures of sharpness, dynamic range, contrast, and whitebalance. Optionally, if bad parts are near edges, a degree of overlapcan be changed to improve the assembled panoramic image.

Another exemplary back end feature employed in some embodiments of theinvention comprises applying pre storage processing to frames of apanoramic image. In an exemplary embodiment of the invention, acquiredframes are processed for one or more of projection simulation (e.g.,cylindrical), intensity, color correction, and registration calculationsprior to storage in memory. Optionally, processing is applied that makesthe frames more similar to one another so that the assembled panoramawill have a uniform appearance across a panoramic range. Optionally,preprocessing contributes to an increase in efficiency. In an exemplaryembodiment of the invention, efficiency increases as preservation ofdata which might normally be lost during the “save to memory” processincreases.

A variety of numerical indicators have been utilized in the abovedescription. It should be understood that these numerical indicatorscould vary even further based upon a variety of engineering principles,materials, intended use and designs incorporated into the invention.Alternatively, or additionally, as available computing capacity and/ormemory in mobile phone handsets become greater, numerical constraintsdescribed hereinabove may be loosened. Additionally, components and/oractions ascribed to exemplary embodiments of the invention and depictedas a single unit may be divided into subunits. Conversely, componentsand/or actions ascribed to exemplary embodiments of the invention anddepicted as sub-units may be combined into a single unit with thedescribed/depicted function.

Alternatively, or additionally, features used to describe a method canbe used to characterize an apparatus and features used to describe anapparatus can be used to characterize a method.

Alternatively, or additionally, various methods described hereinabovecan optionally be provided as software or firmware or integrated intomobile phone handset hardware.

It should be further understood that the individual features describedhereinabove can be combined in all possible combinations andsub-combinations to produce exemplary embodiments of the invention. Theexamples given above are purely illustrative and are not intended tolimit the scope of the invention which is defined solely by thefollowing claims.

The terms “include”, “comprise” and “have” and their conjugates as usedherein mean “including but not necessarily limited to”.

The terms “select”, “selection”, “choose” and “indicate” and theirconjugates as used herein refer to selection from a menu containingicons and/or text as is commonly practiced in computerized interfaces.

The invention claimed is:
 1. A method of assembling a panoramic imagefrom a plurality of frames in a camera, the method comprising: (a)acquiring a video stream using a camera; (b) selecting an initial framefrom the video stream for capture as a still image; (c) detecting amotion direction of the camera while the camera captures the videostream; (d) presenting at least one direction indicator comprising atleast one of a border of a next frame or a center locator of a nextframe, said next frame based on an analysis of at least one frame ofsaid video stream according to the motion direction of said camera forguiding the user to move the camera to said border of said next frame orsaid center locator of said next frame in accordance with said at leastone direction indicator in order to further select said next framedefining at least one additional frame from the video stream for captureas at least one other still image; and, (e) stitching the at least oneadditional frame to the selected initial frame in the camera to create apanoramic image.
 2. A method according to claim 1, wherein the camera isinstalled in a mobile phone handset.
 3. The method according to claim 1,wherein the further selecting includes overlay alignment.
 4. The methodaccording to claim 1, wherein the detection comprises detecting anabsence of motion and triggers acquisition of an additional frame. 5.The method according to claim 1, wherein the selecting is performedmanually by a user.
 6. The method according to claim 1, comprisingdetection of at least one undesirable feature.
 7. The method accordingto claim 6, wherein the undesirable feature comprises a face.
 8. Themethod according to claim 1, wherein the further selecting is performedautomatically by the camera.
 9. The method according to claim 1, whereinthe panoramic image is of a 3D surface.
 10. The method of claim 1,wherein the still image and the at least one another still image have ahigher resolution than the image resolution of the captured videostream.
 11. The method of claim 1, wherein said generated at least onedirection indicator is selected from the group consisting of: arrowindicator, frame indicator, next frame border indicator, next framecenter indicator, blinking indicators, colored indicators, audibledirections and combinations thereof.
 12. A method of assembling apanoramic image using a camera, the method comprising: (a) initializinga process of acquiring a plurality of frames from a video stream forcreating a panoramic image using the camera; (b) detecting a motiondirection of the camera while the camera captures the plurality offrames from the video stream; (c) presenting at least one directionindicator in said video stream comprising at least one of a border ofsaid next frame or a center locator of said next frame, said next framebased on an analysis of at least one previous frame of said video streamaccording to the motion direction of said camera; (d) interrupting theacquiring of the plurality of frames from the video stream according tothe at least one direction indicator; (e) detecting a change in themotion direction of the camera while the camera captures the pluralityof frames from the video stream; (f) resuming acquisition of theplurality of frames from the video stream belonging to the panoramicimage according to the change; and, (g) assembling the acquired framesfrom the video stream to create the panoramic image.
 13. The methodaccording to claim 12, wherein the camera is installed in a mobile phonehandset.
 14. The method according to claim 12, further comprising:excluding from the panoramic image at least one frame of the pluralityof frames, the at least one frame contains an interfering object. 15.The method according to claim 12, wherein the at least one frame notbelonging to the panoramic image is acquired from a different position.16. The method of claim 12, wherein said created panoramic image is atwo dimensional panoramic image.
 17. A camera adapted to assemble apanoramic image from a plurality of frames, the camera comprising: (a) avideo acquisition module for providing a video stream; (b) a selectormodule adapted to select an initial frame from the video stream forcapture as a still image, and to select a next frame from the videostream for capture as a still image; (c) a motion direction detectoradapted to detect the motion direction of the camera while the cameracaptures the video stream, and to present at least one directionindicator in said video stream comprising at least one of a border ofsaid next frame or a center locator of said next frame, said next framebased on an analysis of said initial frame of said video stream; (d) auser interface adapted to guide a user to move the camera in order toselect said next frame from the video stream for capture as a stillimage according to said at least one direction indicator; and, (e) astitching module adapted to stitch said next frame to the initial frameto create a panoramic image.
 18. The camera according to claim 17,provided as part of a mobile phone handset.
 19. A camera comprisingcircuitry adapted to assemble a panoramic image, the camera comprising:(a) a frame acquisition module for acquiring frames from a video stream;(b) a memory with plurality of frames stored therein, only some of theframes from the video stream and belonging to a panoramic imageassociated with the video stream; (c) a camera motion (CaMo) enginewhich detects a motion direction of the camera while the camera capturesthe video stream, and presents at least one direction indicator in saidvideo stream comprising at least one of a border of a next frame or acenter locator of a next frame, said next frame based on an analysis ofat least one previous frame of said video stream according to the motiondirection of said camera; and, (d) a stitching engine adapted toidentify those frames in the memory which belong to the panoramic imageaccording to the at least one direction indicator and assemble theidentified frames from the video stream in sequence.
 20. A cameraaccording to claim 19, provided as part of a mobile phone handset.